Repository navigation
Conversation
b558470 to
cfc6857
Compare
Aman-Mittal
left a comment
There was a problem hiding this comment.
Reviewed the top commit. The claim design is right: an insert against a unique constraint is the only way to settle a race between concurrent requests, and catching the violation rather than pre-checking is what makes it sound. Two inline questions, one about the retry around claim, one about the store surface step 1 introduced.
A few things I checked and found correct, worth recording so nobody re-derives them:
- The text comparison in
matchesStoredRequestis safe.requestandresponsearetext, notjsonb, so nothing normalises key order or numeric scale behind your back, andJsonNodeWritingConverterwrites with the same injected mapper you compare against. Onjsonbthis would have compared re-ordered keys and refused every legitimate repeat. - The migration is portable.
left(),concat()andROW_NUMBER() OVERall exist on PostgreSQL, MySQL and MariaDB, and the derived table does avoid MySQL error 1093. - Partitioning the dedup by
idempotency_keyalone is correct here, because every pre-existing row gets''for the new distinctor in the preceding changeset, so key and (key, distinctor) partition identically. That is only true at this point in history, which is worth the comment it already has. - Nullable
idempotency_keyunder the unique constraint still permits many rows with no key, on all three databases.
One request on the changelog: the comment on command-00007 explains why the non-unique index from 0002 is deliberately left in place, which is good, but nothing tracks the removal. Could that get a JIRA ticket referenced in the comment, so the redundant index does not outlive everyone who remembers why it is there?
Command hooks now receive a CommandContext - the command, its response, its error and its state - instead of separate arguments, so a before hook can mark a command as handled (skipExecution) and have the dispatcher return a response it supplied, and can stop the remaining hooks (skipHooks). The synchronous, async and disruptor dispatchers all honour it; the disruptor dispatcher hands each command a fresh future before publishing its slot. Each command module gets constants for its properties, and the hook switches move under the module they belong to (fineract.command.audit.hook-*-enabled, fineract.command.hook-*-enabled). The audit module reports at startup which of its hooks are on. The JDBC store is renamed to its Jdbc*/Json* types and stores a CommandContext. The webhook command hooks are adapted to the new hook interface. Step 1 of 9 of FINERACT-2485. Co-authored-by: Aleksandar Vidakovic <aleks@apache.org>
…ormat The command store runs on Jackson 3 while Jersey can only run on Jackson 2 (jersey-media-json-jackson 4.0.2 has no Jackson 3 support). Jackson3ObjectMapperConfig defines the application's Jackson 3 mapper to produce the same JSON as the web mapper: the same date, time, ExternalId and Month formats, NON_NULL inclusion, and the two date-time defaults Jackson 3 turned off (WRITE_DATES_AS_TIMESTAMPS, WRITE_DURATIONS_AS_TIMESTAMPS). java.time.Month is written by name, because Jackson 3 writes it as a number that a Jackson 2 client reads back as the wrong month. Credential masking is reapplied: the Jackson 2 MaskedValueSerializer annotation is invisible to Jackson 3. JacksonWireFormatEquivalenceTest pins the agreement, and Jackson3ObjectMapperWiringTest checks that this mapper, not Spring Boot's default, is the one injected. Step 2 of 9 of FINERACT-2485.
UpdateWorkingDaysCommandHandler was the only command handler executing without a retry, so it gave up on the first contention where every other command tries again. It gets its own commandWorkingDaysUpdate instance, configured like the others. The product mix and floating rate handlers named retry instances that were never configured, so they fell back to resilience4j's defaults and retried on every exception. Their instances are configured like the others, and the unused commandMixTaxonomyMappingUpdate instance, left behind when the mix module was removed, is dropped. CommandHandlerRetryCoverageTest walks every CommandHandler on the classpath and fails if one is added without a retry, or names a retry instance that is not configured. Step 3 of 9 of FINERACT-2485.
Every command declares its entity and action names, in the vocabulary of m_permission, and m_command stores them in two new NOT NULL columns (rows written before are backfilled as UNKNOWN). The audit search and the checker inbox are defined in these terms, and maker-checker derives the approval permission from them, so a row without them is invisible to both. Declared as abstract methods on Command rather than an optional interface: a command that forgot them would be stored with none. The names follow m_permission, which is not always what the class name suggests (CACHE/UPDATE, BUSINESS_DATE, COLLATERAL_PRODUCT, CHARTSLAB, MEETING for attendance); notes and images compute the entity from their payload. HookableCommand becomes a marker, and a command read back from the store is a StoredCommand carrying the recorded names. Step 4 of 9 of FINERACT-2485.
The command store can now claim an idempotency key atomically and report what was stored for it, which the idempotency module in the next step builds on. - m_command.custom_idempotency_distinctor (NOT NULL, default '') holds what a key is unique together with, and a unique constraint on (idempotency_key, custom_idempotency_distinctor) makes a claim atomic. Rows from earlier builds that share a key keep the earliest one; the others get their id appended, so no record is deleted. - m_command.response_status_code records the status of an answer that is not the command's own result (a failure, or an answer that is not JSON). - CommandStore gains claim(), getReplayByKey(), matchesStoredRequest() and recordResponse(); every lookup by key also takes the distinctor, and IdempotencyKeyLookupCoverageTest fails the build on one that does not. A stored result that cannot be read back fails loudly instead of answering null. - Writes of the command row are retried on transient and recoverable data-access failures; a constraint violation is not retried, because the row is already there. Until the idempotency module is added, a second request with a key that is already stored is rejected by the constraint instead of being executed again. Step 5 of 9 of FINERACT-2485. Co-authored-by: Aleksandar Vidakovic <aleks@apache.org>
cfc6857 to
14748f6
Compare
Step 5 of 9 of FINERACT-2485 — splitting #6871 into reviewable pieces. Merge in step order.
This PR targets
develop, so until the earlier steps merge it also shows their commits. Review only the top commit — the changes of this step: compare with step 4. Depends on #6910.What this step does
The command store learns to claim an idempotency key atomically and to report what was stored for it. Nothing calls this yet; step 7 does.
0004_idempotency_key_unique.xml: addscustom_idempotency_distinctor(NOT NULL, default'', what a key is unique together with) and a unique constraint on(idempotency_key, custom_idempotency_distinctor). Rows from earlier builds sharing a key keep the earliest; the others get their id appended — no record is deleted.0005_command_response_cache.xml:response_status_code, for an answer that is not the command's own result.CommandStoregainsclaim(),getReplayByKey(),matchesStoredRequest()andrecordResponse(); every lookup by key also takes the distinctor (IdempotencyKeyLookupCoverageTestfails the build otherwise). A stored result that cannot be read back fails loudly instead of returningnull.commandStoreretry instance changes.Interim behaviour until step 7 merges: a second request carrying a key that is already stored is rejected by the constraint instead of being executed again.
The claim/unique-key design comes from #6231 by Aleksandar Vidakovic (credited as co-author).
Verified
fineract-command-jdbcbuild with tests, including the migration test on PostgreSQL, MySQL and MariaDB (Testcontainers) and the store tests; Liquibase registration and DDL-safety checks. Not verified on a running Fineract instance.Stack
Umbrella and history: #6871.